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Aimed at developing methods, tools and techniques for delivery and evaluation of an 
educational module, this article addresses the needs of academicians restructuring, 
adjusting and altering their curricula to meet European Higher Education Area 
standards. The paper focuses on two pressing issues - the applicability of and 
adjustment to European standards in the Russian higher education system. These issues 
are exemplified by the pilot implementation of a course in Language Assessment at 
Kazan (Volga region) Federal University (KFU). In the paper the local impact of the 
course at KFU is viewed at four levels: Reactions, Learning Changes, Behaviour and 
Results. Impact data collected at KFU include the following: end of session written 
feedback, pre- and post-course questionnaires, observation in the classroom, interviews, 
concept maps, teacher portfolios, written assignments, tests/examinations and 
participant journal entries. Viewed as the first step in conducting a full Student Needs 
Analysis, the research is intended to inform the design and delivery of Language 
Assessment courses for graduates majoring in English, Linguistics or Pedagogy 
elsewhere. The methods, techniques and tools developed by the authors may also be 
adapted for application to any University course during piloting, or following its 
introduction. 

Keywords : course delivery, evaluation, Language Assessment, Education Standards, 
diagnostic assessment, formative assessment, EGE, FCE 

INTRODUCTION 

The implications for the Russian higher education system of joining the Bologna 
process were fundamental. They have included the implementation of three-level 
higher vocational education, substantial changes to course content, new methods 
and approaches and new quality assurance systems. Since 1992 the modernization 
of the Russian education system has been a matter of public debate: particularly in 
relation to whether it is preferable to import and reproduce European models or to 
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build a reformed and internationally competitive academic infrastructure on 
established Russian foundations. 

The authors of'The Bologna Process and its Implications for Russia’ (2005] argue 
that 'the Bologna process challenges the Russian state on three levels: the economy, 
society and culture, and state power'. The second component - culture - is 
extremely important in the Russian EFL educational context. To many involved in 
EFL training in the 1990s and even in the 2000s the Bologna changes promised 
academic mobility, and 'the possibility to increase the international and intercultural 
knowledge and skills of students and promote research which addresses 
interdependence (cultural, economical, environmental, political] among nations' 
(Mayor, 1989], Gradually EFL pedagogy textbooks in Russian Universities were 
changed and an EFL training was adopted that would be more in line with 
international practice. What is still lacking, however, is the adoption of current 
approaches to language assessment. While teaching has been reformed, assessment 
has largely remained the same. 

On the principle that knowledge of school curricula, class management, 
methodologies, education theories and assessment ought to be embedded in a wider 
awareness of the impact of educational aims (Darling-Hammond & Bransford, 2005; 
Kalayci, 2012; Rathert, 2012], a project was instituted by a Consortium of 17 
partners from the Russian Federation in cooperation with the Russian Ministry of 
Education and Science. The partners included Cambridge University Press 
Representative Office in Moscow, 14 Russian and 4 EU (Technische Universitat 
Dresden, Germany, University of South Bohemia in Ceske Budejovice, the Czech 
Republic, Southampton Solent University, UK, University of Bedfordshire, UK] 
Universities. Funding was provided by the European Union TEMPUS programme. 
Drawing on international models for training in language assessment (Fulcher, 
2010; Green, 2013; Hughes, 2003], the project, working under the title of'Promoting 
Sustainable Excellence in Testing (ProSET]' set out to address the need for 
assessment reform and to contribute to the development of the Russian National 
Qualifications Framework by promoting assessment literacy among school teachers 
of English. The ProSET consortium developed open-access course materials for 
training teachers in language testing and assessment, which can be downloaded 
fromwww.proset-tempus.net. 

LITERATURE REVIEW 

The authors of The Bologna Process and its Implications for Russia (2005] argue 
that 

... if one seriously considers the idea that training process should be 
centered on student and his/her interests, then the main point in 
evaluation of educational activity should be not training process planning 
and implementation including content of training plans, content and 
timing of disciplines, learning progress but learning outcomes : knowledge 
acquired by student, skills acquired in the course of learning disciplines, 
extension of disciplines in pre-determined field; professional success (at 
employment and career progress]. 

It is not the aim of this paper to focus on all the meanings attached to the term 
'learning outcomes.' Instead, our intent is to acknowledge the richness but also 
importance of the term. Learning outcomes are defined by EFL practitioners as 
‘statements that describe significant and essential learning that learners have 
achieved, and can reliably demonstrate at the end of a course or program’ (Lesch, 
1995].’ In other words, 'learning outcomes identify what the learner will know and 
be able to do by the end of a course or program’ or 'what people have learnt as a 
result of an experience.' Learning outcomes refer to observable and measurable 
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knowledge, skills, attitudes. Learning outcomes are also used to assess the learning 
which has taken place (Lesch, 1995], 

METHODS AND MATERIALS 

Data and context 

At Kazan Federal University (KFU], pilot versions of the course were taken by 27 
undergraduate students majoring in English and Education (Autumn, 2012], 42 
undergraduate students (Autumn, 2013], 7 graduates (Spring, 2013], A further 15 
graduates signed up for a version of the course delivered by the authors in autumn 
2013 at Kzyl Orda State University, Kazakhstan. The students' English language 
proficiency (assessed using the Quick Oxford Placement Test] varied from B1 to Cl 
of the Common European Framework of Reference for Languages (Council of Europe 
2001], The course was delivered in English alongside courses in Theoretical 
Linguistics. 

The aims of the course were explained to the participants as follows: to introduce 
the fundamental principles of Language Assessment and to develop skills in the 
design and validation of testing instruments. 

The project partners identified the following challenges: 

• the prior lack of any testing and assessment component in teacher 
training in the Russian Federation 

• a lack of know-how among teacher trainers 

• a lack of experience among secondary school teachers 

• a lack of awareness of principles underlying the design and development 
of the National English language Test (EGE], used in the Russian 
Federation for matriculation or school leaving purposes. 

• a lack of coordination between universities responsible for teacher 
training 

• equipment and resource shortages (mainly related to the availability of 
literature on assessment]. 

It was concluded that the lack of awareness of assessment issues may lead 
teachers towards inappropriate test preparation activities, which run counter to the 
inventions of the state secondary school curriculum (see Green, 2014], 

Challenges for the first course delivery at the Pre-course Stage at KFU included: 
differentiating groups of students based on their (1] English language proficiency 
and (2] individual course goals: formulating goals for different groups of students; 
identifying the appropriate level of independence for students at a given level. 

An additional complicating factor emerged at the very beginning of course 
delivery, while discussing the topic of standardized testing and the idea of 
standardizing. The Russian Dictionary (1999] defines two senses of the words 
'standartizaciya' (standardizing as a process and a result], standartizirovat' (to 
standardize] as follows: a. making things of the same type all have the same basic 
features; b. destroying individuality and originality in someone/something, while 
Cambridge Advanced Dictionary (CAD] registers only the first meaning of the 
corresponding English verb. The fixed semantic components of the Russian words 
registered in the Dictionary reflect the dominant views and attitudes of the Russian 
ethnos towards this particular referent - standardization. The process of making 
things the same is considered by Russians undesirable and unpleasant, and these 
constitute a markedly pejorative 'common ground' (Fairclough, 2003], The fixed 
values surrounding the idea are imposed on individuals and predetermine their 
textual worlds, discursive practices and identities (De Fina et al, 2006], 

At the end of Module 2 after the class on Unit 6 - Standard Setting, the students 
(27 undergraduates and 15 graduates] participated in a course evaluation survey 
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developed by KFU Philology Department. The survey included a question on 
students' changes in views: "Did the course change your views on something? How?" 
About 97% of the participants answered that they had changed their views on EGE 
and other standardized tests and had begun viewing them more positively. 

The lack of experience in taking different types of international and/or high- 
stakes exams (IELTS, TOEFL, Cambridge ESOL, EGE] affected the accessibility of a 
number of Units of Module 1 (Assessing Reading, Listening, Speaking, Writing, 
Vocabulary and Grammar] and especially Unit 5 of Module 2 (washback, feedback 
and consequences]. The courses also made unfamiliar demands on students in terms 
of learning style: among minor issues slowing the process of learning down were 
underdeveloped team work skills and ineffective or slow communication in English. 

Tools and techniques 

The impact of the course was assessed by the authors at four levels: reactions, 
learning changes, behaviour, results and include feelings immediately after training, 
changes in beliefs, knowledge and skills, application of new ideas over time, effect on 
the organization (KFU], 

The collection of the impact data was conducted with the following tools: end of 
session written feedback, pre- and post-course questionnaires, observation in the 
classroom, interviews, written assignments, tests/examinations, using a database of 
over 2,000 files from Autumn, 2012, 2013 and Spring, 2013. As the process of 
developing, delivering, implementing, and evaluating the course progressed, formal 
inquiry continued through discussion and questionnaires, interviews, and 
observation. 

All the course participants were asked to provide end of course written feedback 
to evaluate the content, design, instructors], delivery and results and to share 
recommendations on the improving the course. Aspects of the course were 
represented by statements, which students rated on a five-point scale: 1 = 'strongly 
disagree’ (the lowest, most negative impression], 3 = 'neither agree nor disagree’ 
(neutral], 5 = 'strongly agree’ (the highest, most positive impression]. 

The pre- and post-course questionnaires designed by the course instructor were 
aimed at comparing participants' views on the main ideas in language testing 
(Validity, Standardization, Item Analysis etc.] before and after course delivery and 
contained predominantly direct questions: e.g. What is the Assessment cycle? What 
are test specifications? etc. Observation in the classroom, written assignments and 
progress tests were held by the course instructor throughout the course. 

The learner-centered approach practiced during the course delivery 
demonstrated the use of ongoing inquiry into the effectiveness and appropriateness 
of the course topics, with resulting information incorporated into the course 
immediately. 

RESULTS 

Feelings after training 

About 67% of the course trainees answering the question 'What was new in the 
course you attended?’ in the post-course questionnaire wrote that ... 'the subject 
[assessment] itself was new and useful for my future career of teacher, ... it gives the 
inside view of the process of marking examination papers and how [the assessment] 
cycle really works'. 

The majority found the language of the course 'quite technical' and testified to 
their inability at the beginning of the course to follow the ideas in the assigned pre¬ 
course reading. 
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Changes in beliefs 

End of session written feedback brought to light the changes the course brought 
about in the views of the course graduates. The four most important issues 
(mentioned in 54% of the evaluation sheets] were:l. 'Tests are to be standardised'. 
2. 'Test development should be taught'. 3. 'Tests are to be pragmatic'. 4. 'Tests are to 
be valid’. 

The idea that 'Education managers need to be taught Principles of Language 
Assessments' was among the most popular in the post-course questionnaires: 47% 
course graduates chose this option. 

Knowledge and skills 

Knowledge and Skills acquired by the students were presented and registered on 
the Unit level. Below is a fragment of a pre-prepared template on Assessing Reading. 
The Blank format was provided by the Project coordinators and used by Russian 
partners while delivering the course. 

DISCUSSION 

When, in 2013, the first course participants reached the point of selecting their 
Masters dissertation topics three of the fourteen students chose topics related to 
Language Assessment. By May 2014 all the three graduates had finished their 
comparative research on different features of Russian Unified State Exam [EGE] and 
Cambridge English: First [FCE] Reading test texts. 

The tools they applied were: a. Flesch Reading Ease; b. Coh-Metrix. Texts of both 
English exams - EGE and FCE - demonstrated correspondence to grade 6 and very 
similar Means of Flesh Reading Ease (EGE Mean is 78.25; Cambridge English First 
Mean is 71.06] which correspond to band 5 in IELTS scale MODEST USER. 

They carried out Coh-Metrix analyses on a corpus of 12 texts (6 FCE and 6 EGE] 
which demonstrated the profiles of the text studied. 

Their research showed that EGE and FCE texts are similar in all the parameters 
applied, though EGE texts selected (and probably adapted to fit the specifications of 
the test] by Russian item writers have greater narrativity, they are simpler in terms 
of syntax, but lower in both deep and referential cohesion (suggesting that a reader 
may face difficulties inferring meanings at the sentence level]. In the EGE test, more 
limited cohesion is balanced by simpler syntax and higher narrativity. The Flesch 
Reading Ease is similar to that found for FCE. 

Table 1. Module 1, Unit 4, Assessing Reading (fragment] Learning Outcomes Assessment (Unit level] _ 

Learning Outcomes Assessed Assessment Description of Weight Submission 

Method Assessment Method % week 

(assignments] 
or length 

_ (exam] _ 

project work group project (3-4 people]: 10 12 

design a reading test task 

for a particular target _ 

group. The test should 10 12 

include a text and at least 
five tasks targeting 
different reading processes. 

Start with the test 25 12 

specifications. 


analyze pros and cons of different 
types of tasks and items aimed at 

assessing reading _ 

analyse reading requirements and 
cognitive features as well as 
grammatical and lexical differences 

across the levels of the CEFR _ 

design items assessing reading at 
different levels: direct vs indirect tests 
on reading. 
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Table 2. Comparative study of EGE and FCE Reading Texts 

Coh-Metrix 

EGE (mean] 

FCE (mean] 

Narrativity 

79 (53-97) 

60.5 (38-71) 

Syntactic Simplicity 

51 (24-80) 

31.5 (18-55) 

Word Concreteness 

61.6 (31-85) 

61.5(22-87) 

Referential Cohesion 

20.1 (6-50) 

26.2(7-88) 

Deep cohesion 

60.6 (19-94) 

70.7 (52-86) 


This experience suggests that the courses were successful in improving the 
assessment literacy of the teacher trainees to the extent that some at least were able 
to engage critically with the national language assessment system and to draw 
international comparisons. 

CONCLUSION 

The current lack of a testing and assessment component in teacher training 
provision in Russia contributes to the absence of transparent, international, 
educational standards in language learning and assessment in Russia. The impact of 
the Language Assessment courses in Kazan Federal University piloted in 2012-2014 
as part of implementation of ProSET was studied by the authors at four levels 
(reactions, learning changes, behaviour and results], A number of positive changes 
were observed in students' views on language assessment. 

As a form of educational innovation, the introduction of the new course needed to 
be managed effectively at the micro-educational level in order to achieve its aims. 
The authors believe that thoughtfully articulated high-level course objectives helped 
to set a high and consistent standard of learning and foster students' creativity. 
Thus, the benefit of articulating clear learning objectives is to identify the 
appropriate level of independence for students at a given level. The interviews and 
evaluation sheets applied demonstrated the trainees' enhanced ability to state an 
independent point of view on different types of exam, including EGE (Unified State 
Examination in English in RF], Success of Language Assessment course to a great 
degree depends on the students sharing the idea that standardizing can be positive 
and that it does not necessarily involve 'destroying individuality and peculiarity'. 
The course also proved to be valuable for making curricular change decisions, 
widening students' horizons, opening channels of communication between 
departments. 
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